Image extraction device, image extraction method, program, and recording medium

ABSTRACT

In the image extraction device, an instruction acquisition unit acquires an instruction input by a user, and an image group selection unit selects a second image group, which has a smaller number of images than a first image group, from the first image group in response to the instruction. Then, an extraction reference determination unit determines an image extraction reference when extracting an image from the second image group based on images included in the first image group, and an image extraction unit extracts one or more images, the number of which is smaller than the number of images in the second image group, from the second image group according to the image extraction reference.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application claims priority under 35 U.S.C. §119 to JapanesePatent Application No. 2015-124693, filed Jun. 22, 2015, all of whichare hereby expressly incorporated by reference into the presentapplication.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image extraction device, an imageextraction method, a program, and a recording medium for automaticallyextracting an image with a high degree of importance for a user from animage group.

2. Description of the Related Art

As analyses of image content, techniques, such as facedetection/expression detection, person recognition, scene detection(night scene, evening scene, sea, nature (green), and the like), andobject detection (pets, food, flowers, train, car, and the like), aregenerally used. In addition, a technique of determining the relationshipbetween persons based on the number of appearances of a person in animage group, the number of persons imaged in the same image, and adistance between the faces of persons is already known. As disclosed inJP2006-236216A, JP2006-79460A, JP2006-81021A, JP4490214B, andJP2014-174787A, an image with a high degree of importance for a user isautomatically extracted from an image group using such techniques.

JP2006-236216A discloses a technique of recognizing a plurality ofpersons in a plurality of images, calculating a degree of intimacybetween persons based on the recognition results, and determining thatpersons recognized as different persons are the same person in a casewhere the difference in the degree of intimacy between personsrecognized as different persons is equal to or less than a referencevalue.

JP2006-79460A discloses a technique of recognizing a plurality ofpersons in a plurality of images, calculating a degree of intimacybetween persons in the plurality of images based on the recognitionresults, and selecting an image, which includes a person having a degreeof intimacy with a viewer equal to or greater than a reference value setin advance, from the plurality of images.

JP2006-81021A and JP4490214B disclose a technique of recognizing aplurality of persons in a plurality of images, calculating a degree ofintimacy in an image, which is the degree of intimacy between persons ineach image, based on the recognition results, calculating a degree ofintimacy in an album, which is the degree of intimacy between persons ina plurality of images, based on the degree of intimacy in an image, andselecting an image, which includes a person having a degree of intimacyin an album with a viewer that is included in a range set in advance,from the plurality of images.

JP2014-174787A discloses a technique of extracting a face image from anumber of images, dragging and dropping a face image of an importantperson into an important region, and dragging and dropping a face imageof a person, who does not need to be included in an electronic album,into an exclusion region.

SUMMARY OF THE INVENTION

The relationship between persons in an image group changes with ahigher-level image group (population) in which the image group isincluded. The following cases (1) and (2) will be described as examples.

(1) In the case of an image group of one year of a family, images mainlyinclude family members (children and parents). In particular, there aremany images of children. Accordingly, it can be estimated that imageswith a strong relationship with children are important.

(2) In the case of an image group of a relative's wedding included in(1), images mainly include a bride and groom. Accordingly, it can beestimated that images with a strong relationship with the bride andgroom are important.

In a case where only each image group of (1) and (2) is taken intoconsideration, important images can be estimated from images included ineach image group as described above based on the features of the imagesincluded in each image group.

However, if it is known that the image group of the relative's weddingof (2) is one of the image groups of one year of the family of (1), itcan be estimated that, also in the image group of the relative's weddingof (2), images including family members appearing frequently in theimage group of one year of the family of (1), which is an image group ofa higher level than the image group of the relative's wedding of (2),are important. Therefore, even in a case where the number of appearancesof the family members in the image group of the wedding of (2) is small,it is possible to preferentially extract images of the family membersfrom the wedding image group of (2).

In order to solve the problems in the related art, it is an object ofthe invention to provide an image extraction device, an image extractionmethod, a program, and a recording medium capable of accuratelyextracting an image with a high degree of importance for a user from animage group.

In order to achieve the aforementioned object, according to an aspect ofthe invention, there is provided an image extraction device comprising:an instruction acquisition unit that acquires an instruction input by auser; an image group selection unit that selects a second image group,which has a smaller number of images than a first image group, from thefirst image group in response to the instruction; an extractionreference determination unit that determines an image extractionreference when extracting an image from the second image group based onimages included in the first image group; and an image extraction unitthat extracts one or more images, the number of which is smaller thanthe number of images in the second image group, from the second imagegroup according to the image extraction reference.

Here, it is preferable to further comprise an image analysis unit thatanalyzes images included in the first image group and calculates arelationship between persons present in the images included in the firstimage group based on the analysis result, and it is preferable that theextraction reference determination unit determines the image extractionreference according to the relationship between persons.

Preferably, the image analysis unit further calculates a relationshipbetween persons present in images included in the second image groupbased on an analysis result of the images included in the second imagegroup, and the extraction reference determination unit furtherdetermines the image extraction reference according to the relationshipbetween persons present in the images included in the second imagegroup.

Preferably, the extraction reference determination unit determines theimage extraction reference by giving priority to the relationshipbetween persons present in the images included in the second image groupover the relationship between persons present in the images included inthe first image group.

Preferably, the image analysis unit comprises: an image informationacquisition section that acquires image information regarding imagesincluded in the first image group; an image feature analysis sectionthat analyzes features of the images included in the first image groupbased on the image information; and a relationship calculation sectionthat calculates a relationship between persons present in the imagesincluded in the first image group based on the image information and theimage features.

Preferably, the relationship calculation section determines a degree ofimportance of each person from the relationship between persons, and theextraction reference determination unit determines the image extractionreference according to the degree of importance of each person.

Preferably, the relationship calculation section determines one or morepersons whose number of times of appearance is equal to or greater thana threshold value in the first image group, among persons present in theimages included in the first image group, as main persons, anddetermines one or more persons who are present in the images included inthe first image group and whose distances from the main persons areequal to or less than a threshold value, among persons other than themain persons who are present in the images included in the first imagegroup, as important persons, and the extraction reference determinationunit determines the image extraction reference according to the mainpersons and the important persons.

Preferably, the relationship calculation section determines one or morepersons whose number of times present in a center position of each imageincluded in the first image group is equal to or greater than athreshold value, among persons appearing a number of times is equal toor greater than a threshold value in the first image group, as the mainpersons.

Preferably, the relationship calculation section determines one or morepersons whose number of times present toward a front direction in eachimage included in the first image group is equal to or greater than athreshold value, among persons appearing a number of times is equal toor greater than a threshold value in the first image group, as the mainpersons.

Preferably, the relationship calculation section determines one or morepersons whose number of times present with a face size equal to orgreater than a threshold value in each image included in the first imagegroup is equal to or greater than a threshold value, among personsappearing a number of times is equal to or greater than a thresholdvalue in the first image group, as the main persons.

Preferably, the relationship calculation section further determines twoor more persons other than the main persons, who are present in a centerposition of each image included in the first image group, as importantpersons.

Preferably, the relationship calculation section further determines twoor more persons other than the main persons, who are present toward afront direction in each image included in the first image group, asimportant persons.

Preferably, the relationship calculation section further determines twoor more persons other than the main persons, who are present with a facesize equal to or greater than a threshold value in each image includedin the first image group, as important persons.

Preferably, the relationship calculation section further calculates arelationship between a person present in each image included in thefirst image group and an object other than the person, and theextraction reference determination unit further determines the imageextraction reference according to the relationship between the personand the object present in each image included in the first image group.

Preferably, the relationship calculation section determines a degree ofimportance of the object from the relationship between the person andthe object, and the extraction reference determination unit determinesthe image extraction reference according to the degree of importance ofthe object.

Preferably, the relationship calculation section determines a scene thathas been imaged a number of times equal to or greater than a thresholdvalue, among scenes of images included in the first image group, as animportant scene, and determines a person and an object, which arepresent in the images included in the first image group and whoserelationship with the important scene is equal to or greater than athreshold value, as an important person and an important object, and theextraction reference determination unit determines the image extractionreference according to the main person, the important person, and theimportant object.

Preferably, the relationship calculation section determines an imagingdate on which imaging has been performed a number of times equal to orgreater than a threshold value, among imaging dates of images includedin the first image group, as an important imaging date, and determines aperson, an object, and a scene which are present in the images includedin the first image group and whose relationship with the importantimaging date is equal to or greater than a threshold value, as animportant person, an important object, and an important scene, and theextraction reference determination unit determines the image extractionreference according to the main person, the important person, theimportant object, and the important scene.

In addition, it is preferable that the imaging date is an imaging timerange, an imaging date, an imaging month, or an imaging season.

Preferably, the relationship calculation section determines an imaginglocation that has been imaged a number of times equal to or greater thana threshold value, among imaging locations of images included in thefirst image group, as an important imaging location, and determines aperson, an object, and a scene which are present in the images includedin the first image group and whose relationship with the importantimaging location is equal to or greater than a threshold value, as animportant person, an important object, and an important scene, and theextraction reference determination unit determines the image extractionreference according to the main person, the important person, theimportant object, and the important scene.

In addition, it is preferable to further include an image groupdetermination unit that determines an image group to be analyzed by theimage analysis unit from the first image group.

Preferably, the instruction acquisition unit acquires an instruction toselect the second image group imaged on an imaging date within apredetermined range input by the user from the first image groupclassified according to imaging dates.

Preferably, the instruction acquisition unit acquires an instruction toselect the second image group imaged on an imaging location within apredetermined range input by the user from the first image groupclassified according to imaging locations.

Preferably, the instruction acquisition unit acquires an instruction toselect the second image group, which is included in a folder within apredetermined range input by the user, from the first image groupclassified according to folders.

In addition, according to another aspect of the invention, there isprovided an image extraction method comprising: a step in which aninstruction acquisition unit acquires an instruction input by a user; astep in which an image group selection unit selects a second imagegroup, which has a smaller number of images than a first image group,from the first image group in response to the instruction; a step inwhich an extraction reference determination unit determines an imageextraction reference when extracting an image from the second imagegroup based on images included in the first image group; and a step inwhich an image extraction unit extracts one or more images, the numberof which is smaller than the number of images in the second image group,from the second image group according to the image extraction reference.

Here, it is preferable to further comprise a step in which an imageanalysis unit analyzes images included in the first image group andcalculates a relationship between persons present in the images includedin the first image group based on the analysis result is furtherincluded, and it is preferable that the extraction referencedetermination unit determines the image extraction reference accordingto the relationship between persons.

Preferably, the step in which the image analysis unit calculates therelationship between persons includes: a step in which an imageinformation acquisition section acquires image information regardingimages included in the first image group; a step in which an imagefeature analysis section analyzes features of the images included in thefirst image group based on the image information; and a step in which arelationship calculation section calculates a relationship betweenpersons present in the images included in the first image group based onthe image information and the image features.

Preferably, the relationship calculation section further calculates arelationship between a person present in each image included in thefirst image group and an object other than the person, and theextraction reference determination unit further determines the imageextraction reference according to the relationship between the personand the object present in each image included in the first image group.

In addition, according to still another aspect of the invention, thereis provided a program causing a computer to execute each step of theimage extraction method described above.

In addition, according to still another aspect of the invention, thereis provided a non-transitory computer-readable recording medium in whicha program causing a computer to execute each step of the imageextraction method described above is recorded.

In the invention, the image extraction reference is determined based onthe images included in the first image group of a higher level than thesecond image group, and images are extracted from the second image groupbased on the image extraction reference. Thus, by determining the imageextraction reference based on the images included in the first imagegroup of a higher level than the second image group, an image with ahigh degree of importance for the user can be accurately extracted fromthe second image group.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an embodiment showing the configuration ofan image extraction device of the invention.

FIG. 2 is a block diagram of an embodiment showing the configuration ofan image analysis unit shown in FIG. 1.

FIG. 3 is a flowchart of an embodiment showing the operation of theimage extraction device of the invention.

FIG. 4 is a conceptual diagram of an example showing an image foracquiring image information.

FIG. 5A is a conceptual diagram of an example showing a case where thedistance between a bride and groom and friends is equal to or less thana threshold value, and FIG. 5B is a conceptual diagram of an exampleshowing a case where the distance between the bride and groom andfriends is greater than the threshold value.

FIG. 6A is a conceptual diagram of an example showing an image in a casewhere two or more persons other than a main person are present towardthe front direction in the center position, and FIG. 6B is a conceptualdiagram of an example showing an image in a case where two or morepersons other than a main person are present with a face size equal toor greater than a threshold value.

FIG. 7A is a conceptual diagram of an example showing a state in whichan image group imaged in 2014 has been selected as a first image group,and FIG. 7B is a conceptual diagram of an example showing a state inwhich an image group imaged in February, 2014 has been selected as asecond image group.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Hereinafter, an image extraction device, an image extraction method, aprogram, and a non-transitory recording medium of the invention will bedescribed in detail based on a preferred embodiment shown in theaccompanying diagrams.

FIG. 1 is a block diagram of an embodiment showing the configuration ofan image extraction device of the invention. An image extraction device10 shown in FIG. 1 automatically extracts an image with a high degree ofimportance for a user from an image group.

The image extraction device 10 includes controlling device such as CPU.The CPU operates as an instruction acquisition unit 12, an image groupselection unit 14, an image analysis unit 16, an extraction referencedetermination unit 18, and an image extraction unit 20.

The instruction acquisition unit 12 acquires various instructions inputby the user, for example, an instruction (instruction of the user) toselect a second image group imaged on an imaging date within apredetermined range from a first image group that is hierarchicallyclassified according to imaging dates, such as year, season, month, day,and time range. Instruction by the user is input by user's operation ofa mouse or touch panel included in the image extraction device 10.

Then, the image group selection unit 14 selects a second image group,which has a smaller number of images than the first image group, fromthe first image group in response to the instruction of the useracquired by the instruction acquisition unit 12.

Here, the first image group is an image group (population) of a higherlevel than the second image group. That is, the second image group is animage group of the lower level than the first image group. Image groupsare classified by imaging dates. For example, in a case where the firstimage group is an image group of 2014, the second image group is animage group of February, 2014, or an image group of Feb. 3, 2014, or animage group of the morning on Feb. 3, 2014, which is an image group ofthe lower level than the image group of 2014. In addition, therelationship between the first and second image groups is relative. Forexample, in a case where the first image group is an image group ofFebruary, 2014, the second image group is an image group of Feb. 3,2014, or an image group of the morning on Feb. 3, 2014, which is animage group of the lower level than the image group of February, 2014.

Then, the image analysis unit 16 analyzes images included in the firstimage group, and calculates the relationship between persons present inthe images included in the first image group based on the analysisresult. Similarly, the image analysis unit 16 calculates therelationship between persons present in images included in the secondimage group based on the analysis result of the images included in thesecond image group.

Here, face detection/expression detection, person recognition, scenedetection (night scene, evening scene, sea, nature (green), and thelike), object detection (pets, food, flowers, train, car, and the like),and the like are included in the image analysis. In addition, therelationship between persons indicates a relationship between a certainperson and another person present in images included in an image group.For example, among persons present in the images included in the firstimage group, a person having a strong relationship with a main personwith the highest degree of importance is a highly important person forthe main person. Therefore, the person having a strong relationship withthe main person is an important person next to the main person.

As shown in FIG. 2, the image analysis unit 16 includes an imageinformation acquisition section 22, an image feature analysis section24, and a relationship calculation section 26.

The image information acquisition section 22 acquires image informationregarding the images included in the first image group.

Here, the image information includes information of imaging dates,scenes (outdoor, mountain, sea, night scene, and the like), imaginglocations (global positioning system (GPS) information), subjects(persons and objects other than persons), and the like. The informationof imaging dates and imaging locations can be acquired from exchangeableimage file format (Exif) information that is accessory information of animage, for example. In addition, the information of scenes, persons, andobjects can be acquired by image analysis, such as scene determination,person recognition, and object recognition.

The image feature analysis section 24 analyzes the features of imagesincluded in the first image group based on the image informationacquired by the image information acquisition section 22.

In this case, a case is also included in which the image extraction unit20 preferentially extracts an image, which is a sea scene and in whichperson A is present. That is, an image which is a sea scene and in whichperson A is present is extracted, but the extraction of other images isnot excluded.

In addition, a case is also included in which the image extraction unit20 specifies an image, which is a sea scene and in which person A ispresent, and extracts only a part of the image. That is, for example,evaluation values of images are calculated based on the image analysisresult of the image analysis unit 16 for persons present in the images,and faces/expressions of the person, scenes, objects, and the like, apredetermined number of images with high evaluation of sea likeness areextracted in order, or a predetermined number of images with highevaluation of the likelihood of person A are extracted in order, or apredetermined number of images with high evaluation of the facialexpression of person A are extracted in order, or a predetermined numberof images having a high overall point of these evaluation values (alsoincluding a case of weighting) are extracted in order.

Here, the image features include features of the imaging content (scene,the number of appearances of a person or an object, and the like),imaging dates (imaging period of the first image group, an imagingfrequency, and the like), an imaging location (the number of times ofimaging in each imaging location and the like), and the like.

The relationship calculation section 26 calculates the relationshipbetween persons, who are present in the images included in the firstimage group, based on the image information acquired by the imageinformation acquisition section 22 and the image features analyzed bythe image feature analysis section 24. In addition, based on the imageinformation and the image features, the relationship calculation section26 calculates the relationship between persons present in the imagesincluded in the second image group.

Then, based on the images included in the first image group, theextraction reference determination unit 18 determines an imageextraction reference when extracting an image from the second imagegroup.

In the case of the present embodiment, the extraction referencedetermination unit 18 determines the image extraction referenceaccording to the relationship between the persons present in the imagesincluded in the first image group and the relationship between thepersons present in the images included in the second image group, whichhave been calculated by the image analysis unit 16.

Finally, the image extraction unit 20 extracts one or more images(extracted image), the number of which is smaller than the number ofimages in the second image group, from the second image group accordingto the image extraction reference determined by the extraction referencedetermination unit 18.

Here, the image extraction reference includes a scene, a person, anobject, and the like. For example, in a case where the image extractionreference is an image which is a sea scene and in which person A ispresent, an image which is a sea scene and in which person A is presentis extracted from the second image group.

Next, the operation of the image extraction device 10 will be describedwith reference to the flowchart shown in FIG. 3.

In the image extraction device 10, an instruction to select a secondimage group from a first image group input by the user is acquired firstby the instruction acquisition unit 12 (step S1). In response to theinstruction, the image group selection unit 14 selects a second imagegroup from the first image group (step S2).

Then, the image analysis unit 16 analyzes each image included in thefirst image group, and calculates the relationship between personspresent in the images included in the first image group based on theanalysis result of the images included in the first image group.Similarly, the image analysis unit 16 calculates the relationshipbetween persons present in images included in the second image groupbased on the analysis result of the images included in the second imagegroup (step S3).

In the image analysis unit 16, the image information acquisition section22 acquires image information regarding the images included in the firstimage group, and the image feature analysis section 24 analyzes thefeatures of the images included in the first image group based on theimage information. Then, based on the image information and the imagefeatures, the relationship calculation section 26 calculates therelationship between the persons present in the images included in thefirst image group, and calculates the relationship between the personspresent in the images included in the second image group.

Then, according to the relationship between the persons present in theimages included in the first image group and the relationship betweenthe persons present in the images included in the second image group,the extraction reference determination unit 18 determines an imageextraction reference when extracting an image from the second imagegroup (step S4).

Finally, the image extraction unit 20 extracts an image from the secondimage group according to the image extraction reference (step S5).

In the image extraction device 10, an image extraction reference isdetermined in consideration of not only the relationship between thepersons present in the images included in the second image group butalso the relationship between the persons present in the images includedin the first image group, and an image is extracted from the secondimage group based on the image extraction reference. Thus, bydetermining the image extraction reference based on the images includedin the first image group of a higher level than the second image group,an image with a high degree of importance for the user can be accuratelyextracted from the second image group.

Next, the image information acquisition section 22, the image featureanalysis section 24, and the relationship calculation section 26provided in the image analysis unit 16 will be further described.

Table 1 shows an example of image information obtained from an imageshown in FIG. 4 by the image information acquisition section 22.

TABLE 1 Image information Imaging date Monday, Apr. 9, 2015 SceneOutdoor Imaging location xxx (GPS information) Subject Person A Regionin an image Distance from another subject and Bicycle: positionalrelationship with another . . . subject Bicycle . . .

In the case of the image shown in FIG. 4, as shown in Table 1, the imageinformation acquisition section 22 acquires, for example, “imaging dateis Monday, Apr. 9, 2015” and “imaging location is xxx (GPS information)”from the Exif information as image information.

In addition, by image analysis, for example, “scene is outdoor”,“subjects are a person A and a bicycle”, and the like are acquired. Forthe person A, for example, as a region in an image (in which locationand size in the image the person A is present?) or a distance fromanother subject and the positional relationship, a distance from thebicycle, positional relationship, and the like are acquired as imageinformation. For the bicycle, image information is similarly acquired.

The image information acquisition section 22 acquires theabove-described image information from each image included in the firstimage group.

Subsequently, Table 2 shows an example of image features obtained byanalyzing each image included in the first image group based on theimage information by the image feature analysis section 24. As shown inTable 2, the image features include image features for the imagingcontent, imaging date and time, imaging location, and the like of eachimage.

TABLE 2 Items Image features Imaging Persons appearing frequently:person A, person B, person C, . . . content Objects appearingfrequently: dogs, cats, food, . . . Scenes appearing frequently:outdoors, indoor, mountain, sea, . . . . . . Imaging Image group ofseveral months: several images captured almost date every day Imagegroup of one year: a large number of images captured once in a fewmonths Image group of one day: a large number of images captured duringseveral hours . . . Imaging Imaging only in the vicinity of a particularlocation location Imaging in various locations . . . . . . . . .

As image features of the imaging content, as shown in Table 2, forexample, “persons appearing frequently (equal to or greater than athreshold value) in the first image group are person A, person B, personC, . . . ”, “objects appearing frequently are dogs, cats, food, . . . ”,and “scenes appearing frequently are outdoor, indoor, mountain, sea, . .. ” are analyzed by the image feature analysis section 24.

As image features of the imaging date, for example, “first image groupis an image group of several months and several images have beencaptured almost every day (images are captured on a daily basis)”,“first image group is an image group of one year and a large number ofimages have been captured once in a few months (images are captured onlyin a big event, such as a trip)”, and “first image group is an imagegroup of one day and a large number of images have been captured duringseveral hours (images are captured in a particular event, such as awedding)” are analyzed.

As image features of the imaging location, for example, “imaging hasbeen performed only in the vicinity of a particular location” and“imaging has been performed in various locations” are analyzed. Theimage feature analysis section 24 analyzes the above-described imagefeatures from the image information of each image included in the firstimage group.

Then, in the relationship calculation section 26, for example, therelationship between persons present in the images included in the firstimage group is calculated. Based on the relationship, it is possible todetermine the degree of importance of each person, such as a main personor an important person.

For example, the relationship calculation section 26 can determine oneor more persons appearing frequently (equal to or greater than athreshold value) in the first image group, among the persons present inthe images included in the first image group, as main persons based onthe relationship between the persons.

In addition, the relationship calculation section 26 can determine amain person based on a combination of the number of appearances in thefirst image group and the number of times that each person is present inthe center position of each image included in the first image group. Inthis case, the relationship calculation section 26 determines, forexample, a person whose number of times present in the center positionof each image included in the first image group is equal to or greaterthan a threshold value, among persons whose number of time of appearingis equal to or greater than a threshold value in the first image group,as a main person.

In addition, the relationship calculation section 26 can determine amain person based on a combination of the number of appearances in thefirst image group and the number of times that each person is presenttoward the front direction in the images included in the first imagegroup. In this case, the relationship calculation section 26 determines,for example, a person whose number of times present toward the frontdirection in the images included in the first image group is equal to orgreater than a threshold value, among persons whose number of times ofappearance is equal to or greater than a threshold value in the firstimage group, as a main person.

In addition, the relationship calculation section 26 can determine amain person based on a combination of the number of appearances in thefirst image group and the number of times that each person is present inthe images included in the first image group with a face size is equalto or greater than a threshold value (including the case of zoomimaging). In this case, the relationship calculation section 26determines, for example, a person whose number of times present in theimages included in the first image group with a face size equal to orgreater than a threshold is equal to or greater than a threshold value,among persons whose number of times of appearance is equal to or greaterthan a threshold value in the first image group, as a main person.

In addition, the relationship calculation section 26 can determine amain person based on a combination of the number of appearances in thefirst image group and at least one of the number of times that eachperson is present in the center position of each image included in thefirst image group, the number of times by which each person is presenttoward the front direction in the images included in the first imagegroup, and the number of times by which each person is present in theimages included in the first image group with a face size equal to orgreater than a threshold value.

In addition, the relationship calculation section 26 can determine, forexample, one or more persons who are present in the images included inthe first image group and whose distances from a main person is equal toor less than a threshold value, among persons other than the main personwho are present in the images included in the first image group, asimportant persons based on the relationship between persons.

For example, FIGS. 5A and 5B are images captured at a wedding. In theimage shown in FIG. 5A, a bride and groom and four friends of the brideand groom are present. In a case where the bride and groom surrounded bya circle are main persons, for example, four friends surrounded by asquare, of which distances from the bride and groom are equal to or lessthan a threshold value, are determined as important persons. Inaddition, in the image shown in FIG. 5B, there are several persons inaddition to the bride and groom. For example, persons surrounded by atriangle, of which distances from the bride and groom surrounded by acircle are greater than a threshold value, are determined as otherpersons who are neither main persons nor important persons.

In addition, even in a case where a main person is not present in theimages included in the first image group, the relationship calculationsection 26 can determine the degree of importance of each person otherthan the main person based on the relationship between persons otherthan the main person who are present in the images included in the firstimage group.

As surrounded by a circle shown in FIG. 6A, in a case where two or morepersons other than a main person are present in the center position ofeach image included in the first image group, the relationshipcalculation section 26 can determine the two or more persons other thanthe main person as important persons based on the relationship betweenpersons other than the main person.

In addition, as surrounded by a circle shown in FIGS. 6A and 6B, in acase where two or more persons other than a main person are presenttoward the front direction in each image included in the first imagegroup, the relationship calculation section 26 can determine the two ormore persons other than the main person as important persons based onthe relationship between persons other than the main person.

As surrounded by a circle shown in FIG. 6B, in a case where two or morepersons other than a main person are present in each image included inthe first image group with a face size equal to or greater than athreshold value, the relationship calculation section 26 can determinethe two or more persons other than the main person as important personsbased on the relationship between persons other than the main person.

As described above, in a case where a main person and important personsare determined by the relationship calculation section 26, theextraction reference determination unit 18 determines an imageextraction reference according to the main person and the importantpersons, that is, according to the degree of importance of each person.

In addition, the relationship calculation section 26 may calculate therelationship between each person present in the images included in thefirst image group and each object other than the person based on theanalysis result of the images included in the first image group.

In this case, the extraction reference determination unit 18 candetermine the image extraction reference according to the relationshipbetween each person and each object present in the images included inthe first image group.

In addition, the relationship calculation section 26 can determine thedegree of importance of each object for each person, as an importantobject, based on the relationship between each person and each object.

In this case, the extraction reference determination unit 18 candetermine the image extraction reference according to the importantobject, that is, according to the degree of importance of the object.

For example, as in the image shown in the diagram, in a case where abicycle (object) is present near person A (person), it can be estimatedthat person A's hobby is riding a bicycle and a bicycle is important forperson A.

In addition to the relationship between persons, for example, therelationship calculation section 26 can use the relationship between aperson and an object, the relationship between a person and a scene, therelationship between a person and an imaging date, the relationshipbetween a person and an imaging location, the relationship between anobject and an object, and the like based on the images included in thefirst image group.

In the case of using the relationship between a scene and a personpresent in each image included in the first image group, therelationship calculation section 26 determines, for example, a scenethat has been imaged many times (equal to or greater than a thresholdvalue), among the scenes of the images included in the first imagegroup, as an important scene. Then, a person and an object, which arepresent in the images included in the first image group and are highly(equal to or greater than a threshold value) related to an importantscene, are determined as an important person and an important object.

In this case, the extraction reference determination unit 18 candetermine the image extraction reference according to the main person,the important person, and the important object.

For example, in a case where there are many sea images in the firstimage group, it can be estimated that the sea is an important scene.Therefore, a person and an object which are highly related to animportant scene, in the case of this example, a person and an objectimaged in the sea scene are important.

In the case of using the relationship between an imaging date and aperson present in each image included in the first image group, therelationship calculation section 26 determines, for example, an imagingdate on which imaging has been performed many times (equal to or greaterthan a threshold value), among the imaging dates of the images includedin the first image group, as an important imaging date. Then, a person,an object, and a scene, which are present in the images included in thefirst image group and have a high (equal to or greater than a thresholdvalue) relationship with an important imaging date, are determined as animportant person, an important object, and an important scene.

In this case, the extraction reference determination unit 18 candetermine the image extraction reference according to the main person,the important person, the important object, and the important scene.

For example, an imaging date on which imaging has been performed manytimes in the first image group can be estimated to be an importantimaging date. Therefore, a person, an object, and a scene having astrong relationship with an important imaging date are important. Forexample, in an image group of one year, in a case where hundreds ofimages have been captured in a day on trips in summer and winter,persons, objects, and scenes that are present in the images included inthe image group during the trips are important.

As the imaging date, it is possible to use not only the imaging day butalso an imaging time range, an imaging month, an imaging season, and thelike.

In the case of using the relationship between an imaging location and aperson present in each image included in the first image group, therelationship calculation section 26 determines, for example, an imaginglocation that has been imaged many times (equal to or greater than athreshold value), among the imaging locations of the images included inthe first image group, as an important imaging location. Then, a person,an object, and a scene, which are present in the images included in thefirst image group and have a high (equal to or greater than a thresholdvalue) relationship with an important imaging location, are determinedas an important person, an important object, and an important scene.

In this case, the extraction reference determination unit 18 determinesthe image extraction reference according to the main person, theimportant person, the important object, and the important scene.

For example, an imaging location that has been imaged many times in thefirst image group can be estimated to be an important imaging location.Therefore, a person, an object, and a scene having a strong relationshipwith an important imaging location are important. For example, in thefirst image group, in a case where hundreds of images have been capturedin Paris, persons, objects, and scenes that are present in the imagesincluded in the image group, which have been captured in Paris, areimportant.

In a case where a scene, an imaging date, an imaging location, and thelike having a large number of times of imaging are not present in thefirst image group, the relationship calculation section 26 does not needto consider the scenes, imaging dates, imaging locations, and the likeof the images included in the first image group.

Table 3 is an example of a case where the important elements in thefirst image group for a person, a scene, an imaging date, an imaginglocation, . . . are person A, sea, May 10, xx, . . . . In this case, forexample, for a person, an important person is person A. Since person Ais present in an image A, the degree of importance of the image A ishigh. Based on the degree of importance of a person, the degree ofimportance of each image included in the first image group can bedetermined. For example, it can be determined that the degree ofimportance of images B and C is medium and the degree of importance ofan image D is low. This is the same for the scene, the imaging date, theimaging location, and the like.

TABLE 3 Imaging Person Scene Imaging date location . . . ImportantPerson A Sea May 10 xx elements in an image group Image A Large Image BMedium Image C Medium Image D Small . . . . . .

Next, a case where a second image group is selected from a first imagegroup hierarchically classified according to imaging dates will bedescribed as an example.

As shown in FIG. 7A, in the present embodiment, it is assumed that thefirst image group is an image group of 2014 and there is no image groupof a higher level than the image group of 2014. In this case, based onthe relationship between persons present in images included in the imagegroup of 2014, the relationship between the persons present in theimages included in the image group of 2014 is determined. In the presentembodiment, it is assumed that family members (children and parents) aredetermined to be important persons from the relationship between thepersons present in the images included in the image group of 2014.

That is, family images are preferentially extracted from the image groupof 2014, and are displayed as recommended images, for example.

Then, as shown in FIG. 7B, it is assumed that an image group ofFebruary, 2014 has been selected from the image group of 2014, as asecond image group, by the user. In this case, the relationship betweenthe persons present in the images included in the image group ofFebruary, 2014 is determined. In the present embodiment, it is assumedthat, since most of the images are wedding images on Feb. 14, 2014, thebride and groom are determined to be important persons from therelationship between the persons present in the images included in theimage group of February, 2014.

That is, as shown in Table 4, it can be seen that the degree ofimportance of the bride and groom is high in a case where the imagegroup of only February, 2014 is taken into consideration but the degreeof importance of the family is high in a case where the image group ofthe entire 2014 is taken into consideration.

TABLE 4 Imaging date Important persons 2014 All family January FebruaryBride and groom March . . .

In a known image extraction device, since the image group of onlyFebruary, 2014 is taken into consideration, images including the brideand groom are extracted from the image group of February, 2014. For thisreason, even if the first image group is images owned by the familymembers of the bride and groom and images including the family membersof the bride and groom are present in the image group of February, 2014,the images of the family members cannot be extracted from the imagegroup of February, 2014 since the degree of importance of the familymembers is low in the image group of February, 2014.

That is, in the known image extraction device, images of the bride andgroom are preferentially extracted from the image group of February,2014, and are displayed as recommended images, for example.

In contrast, in the image extraction device 10 of the presentembodiment, the image extraction reference is determined taking intoconsideration not only the image group of February, 2014 but also theimage group of the entire 2014 of a higher level than the image group ofFebruary, 2014. Therefore, based on the image extraction reference, inaddition to images in which the bride and groom with a high degree ofimportance are present in the image group of February, 2014, images inwhich family members of the bride and groom with a high degree ofimportance are present in the image group of 2014 of a higher level thanthe image group of February, 2014 can be extracted from the image groupof February, 2014.

That is, in the image extraction device 10 of the present embodiment,images of the bride and groom and their family members arepreferentially extracted from the image group of February, 2014, and aredisplayed as recommended images, for example.

In this case, it is desirable to determine the image extractionreference by giving priority to the relationship between persons (thedegree of importance of a person) present in the images included in theimage group of February, 2014, which is a second image group of thelower level, over the relationship between persons (the degree ofimportance of a person) present in the images included in the imagegroup of 2014 that is a first image group of the higher level. In thismanner, it is possible to extract a number of images of the bride andgroom preferentially from the image group of February, 2014 and toextract a small number of family images from the image group ofFebruary, 2014.

Not only can the user select February, 2014 from the image group of2014, but also the user can select simultaneously a plurality of months,for example, February, March, and June, 2014.

In addition, not only can the user select a second image group from thefirst image group based on the imaging date, but also the user canselect a second image group from the first image group based on theimaging location or a folder.

In a case where a second image group is selected from the first imagegroup based on the imaging location, an instruction to select a secondimage group captured in an imaging location within a predetermined rangeinput by the user from the first image group hierarchically classifiedaccording to imaging locations is acquired by the instructionacquisition unit 12. Then, in response to the user instruction acquiredby the instruction acquisition unit 12, the image group selection unit14 selects an image group of the imaging location of the lower levelthan the first image group, as a second image group, from the firstimage group.

As shown in Table 5, the first image group is an image group captured inthe entire Japan, and is hierarchically classified according to imaginglocations, such as Kyoto (parents' home), Roppongi (workplace), andShinyurigaoka (home). In a case where the second image group is an imagegroup captured in Kyoto (parents' home), the degree of importance ofgrandparents and relatives is high and the degree of importance offamily members (children and parents) is low in a case where the imagegroup of only Kyoto is taken into consideration. However, in a casewhere the image group of the entire Japan is taken into consideration,it is assumed that the degree of importance of family members is high.

TABLE 5 Imaging location Important persons Japan All Family membersKyoto (parents' home) grandparents and relatives Roppongi (workplace)coworkers Shinyurigaoka (home) Family members . . .

Similarly, in a known image extraction device, since the image group ofonly Kyoto (parents' home) is taken into consideration, images includinggrandparents and relatives are extracted from the image group of Kyoto(parents' home), and family images are not extracted.

In contrast, in the image extraction device 10 of the presentembodiment, the image extraction reference is determined taking intoconsideration not only the image group of only Kyoto (parents' home) butalso the image group of the entire Japan of a higher level than theimage group of Kyoto. Therefore, based on the image extractionreference, in addition to images in which grandparents and relativeswith a high degree of importance are present in the image group of Kyoto(parents' home), images in which family members with a high degree ofimportance are present in the image group of the entire Japan of ahigher level than the image group of Kyoto can be extracted from theimage group of Kyoto (parents' home).

In a case where a second image group is selected from the first imagegroup based on a folder, an instruction to select an image group, whichis included in a folder within a predetermined range input by the user,from the first image group classified according to folders is acquiredby the instruction acquisition unit 12. Then, in response to the userinstruction acquired by the instruction acquisition unit 12, the imagegroup selection unit 14 selects an image group of a folder of the lowerlevel than the first image group, as a second image group, from thefirst image group.

As shown in Table 6, the first image group is an image group of theentire root folder, and is hierarchically classified according to afolder of each event, such as a wedding, an athletic meet, and abirthday. In a case where the second image group is an image group of awedding folder, the degree of importance of the bride and groom is highand the degree of importance of family members (children and parents) islow in a case where the image group of only the wedding folder is takeninto consideration. However, in a case where the image group of theentire root folder is taken into consideration, it is assumed that thedegree of importance of family members is high.

TABLE 6 Folder Important persons Root folder All Family members WeddingBride and groom Athletic meet Birthday . . .

Similarly, in a known image extraction device, since the image group ofonly the wedding folder is taken into consideration, images includingthe bride and groom are extracted from the image group of the weddingfolder, and family images are not extracted.

In contrast, in the image extraction device 10 of the presentembodiment, the image extraction reference is determined taking intoconsideration not only the image group of only the wedding folder butalso the image group of the entire root folder of a higher level thanthe image group of the wedding folder. Therefore, based on the imageextraction reference, in addition to images in which the bride and groomwith a high degree of importance are present in the image group of thewedding folder, images in which family members with a high degree ofimportance are present in the image group of the entire root folder of ahigher level than the image group of the wedding folder can be extractedfrom the image group of the wedding folder.

The image extraction device of the invention can be used in the case ofselecting a second image group to be used in a composite image from thefirst image group when creating a composite image, such as an electronicalbum, a photo book, a collage print, and a calendar with an image.

For example, in the case of creating a collage print from an image groupof one month, it is possible to determine an image extraction referencein consideration of the image group of one month and an image group ofthe entire one year of a higher level than the image group of one monthand to extract an image group to be used in a collage print from theimage group of one month based on the image extraction reference. Inaddition, in the case of creating a photo book of 24 pages for 12 months(two-page spread per month) from an image group of one year, it ispossible to determine an image extraction reference in consideration ofan image group of each month and an image group of the entire one yearand to extract an image group to be used in a two-page spread of eachmonth from the image group of each month based on the image extractionreference.

The image extraction device of the invention can be applied to a videoimage without being limited to a still image.

As the first image group, for example, an image group that the useruploads from a server or the like in order to create a composite image,such as an electronic album, a photo book, a collage print, and acalendar with an image, can be exemplified.

Alternatively, in response to the instruction of the user acquired bythe instruction acquisition unit 12, a first image group may be selectedfrom an image group already uploaded by the user. Thus, the user canselect a first image group by himself or herself.

In addition, in a case where a second image group is selected, a firstimage group including the second image group may be automaticallyselected. For example, a case is considered in which a user attends thewedding of a coworker of the company and images of the wedding areuploaded on the wedding day. In this case, an image group obtained bycombining the image group (second image group) uploaded on the weddingday and an arbitrary image group already uploaded before the wedding daymay be automatically selected as a first image group. In this manner,the user can automatically determine the first image group without beingaware of the first image group.

In addition, an image group determination unit may be provided todetermine an image group to be analyzed by the image analysis unit 16from the first image group. For example, in a case where the first imagegroup is an image group of 2014 and an image group of May 6, 2014 isselected as a second image group, for example, an image group of May,2014 or an image group of 2015 can be determined as an image group to beanalyzed by the image analysis unit 16 from the image group of 2014 thatis the first image group. This is the same for a case where the firstimage group is classified according to imaging locations and folders.

It is not essential to determine the image extraction reference based onthe relationship between persons present in the images included in thesecond image group and extract an image from the second image groupbased on the image extraction reference. The image extraction referencemay be determined based on only the relationship between persons presentin the images included in the first image group, and an image may beextracted from the second image group based on the image extractionreference.

In addition, without being limited to the relationship and the degree ofimportance, the extraction reference determination unit 18 may determinethe image extraction reference when extracting an image from the secondimage group based on the images included in the first image group, thatis, using various kinds of information obtained from the images includedin the first image group.

In the device of the invention, each component of the device may beformed using dedicated hardware, or each component may be formed using aprogrammed computer.

The method of the invention can be realized, for example, by a programcausing a computer to execute each step of the method. In addition, itis also possible to provide a computer-readable recording medium in thatthe program is recorded.

While the invention has been described in detail, the invention is notlimited to the above-described embodiment, and various improvements andmodifications may be made without departing from the scope and spirit ofthe invention.

What is claimed is:
 1. An image extraction device, comprising: aninstruction acquisition unit that acquires an instruction input by auser; an image group selection unit that selects a second image group,which has a smaller number of images than a first image group, from thefirst image group in response to the instruction; an extractionreference determination unit that determines an image extractionreference when extracting an image from the second image group based onimages included in the first image group; and an image extraction unitthat extracts one or more images, the number of which is smaller thanthe number of images in the second image group, from the second imagegroup according to the image extraction reference.
 2. The imageextraction device according to claim 1, further comprising: an imageanalysis unit that analyzes images included in the first image group andcalculates a relationship between persons present in the images includedin the first image group based on the analysis result, wherein theextraction reference determination unit determines the image extractionreference according to the relationship between persons.
 3. The imageextraction device according to claim 2, wherein the image analysis unitfurther calculates a relationship between persons present in imagesincluded in the second image group based on an analysis result of theimages included in the second image group, and the extraction referencedetermination unit further determines the image extraction referenceaccording to the relationship between persons present in the imagesincluded in the second image group.
 4. The image extraction deviceaccording to claim 3, wherein the extraction reference determinationunit determines the image extraction reference by giving priority to therelationship between persons present in the images included in thesecond image group over the relationship between persons present in theimages included in the first image group.
 5. The image extraction deviceaccording to claim 2, wherein the image analysis unit comprises: animage information acquisition section that acquires image informationregarding images included in the first image group; an image featureanalysis section that analyzes features of the images included in thefirst image group based on the image information; and a relationshipcalculation section that calculates a relationship between personspresent in the images included in the first image group based on theimage information and the image features.
 6. The image extraction deviceaccording to claim 5, wherein the relationship calculation sectiondetermines a degree of importance of each person from the relationshipbetween persons, and the extraction reference determination unitdetermines the image extraction reference according to the degree ofimportance of each person.
 7. The image extraction device according toclaim 6, wherein the relationship calculation section determines one ormore persons whose number of times of appearance is equal to or greaterthan a threshold value in the first image group, among persons presentin the images included in the first image group, as main persons, anddetermines one or more persons who are present in the images included inthe first image group and whose distances from the main persons areequal to or less than a threshold value, among persons other than themain persons who are present in the images included in the first imagegroup, as important persons, and the extraction reference determinationunit determines the image extraction reference according to the mainpersons and the important persons.
 8. The image extraction deviceaccording to claim 7, wherein the relationship calculation sectiondetermines one or more persons whose number of times present in a centerposition of each image included in the first image group is equal to orgreater than a threshold value, among persons whose number of times ofappearance in the first image group is equal to or greater than athreshold value, as the main persons.
 9. The image extraction deviceaccording to claim 7, wherein the relationship calculation sectiondetermines one or more persons whose number of times present toward afront direction in each image included in the first image group is equalto or greater than a threshold value, among persons whose number oftimes of appearance in the first image group is equal to or greater thana threshold value, as the main persons.
 10. The image extraction deviceaccording to claim 7, wherein the relationship calculation sectiondetermines one or more persons whose number of times present with a facesize equal to or greater than a threshold value in each image includedin the first image group is equal to or greater than a threshold value,among persons whose number of times of appearance in the first imagegroup is equal to or greater than a threshold value, as the mainpersons.
 11. The image extraction device according to claim 7, whereinthe relationship calculation section further determines two or morepersons other than the main persons, who are present in a centerposition of each image included in the first image group, as importantpersons.
 12. The image extraction device according to claim 7, whereinthe relationship calculation section further determines two or morepersons other than the main persons, who are present toward a frontdirection in each image included in the first image group, as importantpersons.
 13. The image extraction device according to claim 7, whereinthe relationship calculation section further determines two or morepersons other than the main persons, who are present with a face sizeequal to or greater than a threshold value in each image included in thefirst image group, as important persons.
 14. The image extraction deviceaccording to claim 7, wherein the relationship calculation sectionfurther calculates a relationship between a person present in each imageincluded in the first image group and an object other than the person,and the extraction reference determination unit further determines theimage extraction reference according to the relationship between theperson and the object present in each image included in the first imagegroup.
 15. The image extraction device according to claim 14, whereinthe relationship calculation section determines a degree of importanceof the object from the relationship between the person and the object,and the extraction reference determination unit determines the imageextraction reference according to the degree of importance of theobject.
 16. The image extraction device according to claim 15, whereinthe relationship calculation section determines a scene that has beenimaged a number of times equal to or greater than a threshold value,among scenes of images included in the first image group, as animportant scene, and determines a person and an object, which arepresent in the images included in the first image group and whoserelationship with the important scene is equal to or greater than athreshold value, as an important person and an important object, and theextraction reference determination unit determines the image extractionreference according to the main person, the important person, and theimportant object.
 17. The image extraction device according to claim 15,wherein the relationship calculation section determines an imaging dateon which imaging has been performed a number of times equal to orgreater than a threshold value, among imaging dates of images includedin the first image group, as an important imaging date, and determines aperson, an object, and a scene which are present in the images includedin the first image group and whose relationship with the importantimaging date is equal to or greater than a threshold value, as animportant person, an important object, and an important scene, and theextraction reference determination unit determines the image extractionreference according to the main person, the important person, theimportant object, and the important scene.
 18. The image extractiondevice according to claim 17, wherein the imaging date is an imagingtime range, an imaging date, an imaging month, or an imaging season. 19.The image extraction device according to claim 15, wherein therelationship calculation section determines an imaging location that hasbeen imaged a number of times equal to or greater than a thresholdvalue, among imaging locations of images included in the first imagegroup, as an important imaging location, and determines a person, anobject, and a scene which are present in the images included in thefirst image group and whose relationship with the important imaginglocation is equal to or greater than a threshold value, as an importantperson, an important object, and an important scene, and the extractionreference determination unit determines the image extraction referenceaccording to the main person, the important person, the importantobject, and the important scene.
 20. The image extraction deviceaccording to claim 2, further comprising: an image group determinationunit that determines an image group to be analyzed by the image analysisunit from the first image group.
 21. The image extraction deviceaccording to claim 1, wherein the instruction acquisition unit acquiresan instruction to select the second image group imaged on an imagingdate within a predetermined range input by the user from the first imagegroup classified according to imaging dates.
 22. The image extractiondevice according to claim 1, wherein the instruction acquisition unitacquires an instruction to select the second image group imaged in animaging location within a predetermined range input by the user from thefirst image group classified according to imaging locations.
 23. Theimage extraction device according to claim 1, wherein the instructionacquisition unit acquires an instruction to select the second imagegroup, which is included in a folder within a predetermined range inputby the user, from the first image group classified according to folders.24. An image extraction method, comprising: a step in which aninstruction acquisition unit acquires an instruction input by a user; astep in which an image group selection unit selects a second imagegroup, which has a smaller number of images than a first image group,from the first image group in response to the instruction; a step inwhich an extraction reference determination unit determines an imageextraction reference when extracting an image from the second imagegroup based on images included in the first image group; and a step inwhich an image extraction unit extracts one or more images, the numberof which is smaller than the number of images in the second image group,from the second image group according to the image extraction reference.25. The image extraction method according to claim 24, furthercomprising: a step in which an image analysis unit analyzes imagesincluded in the first image group and calculates a relationship betweenpersons present in the images included in the first image group based onthe analysis result, wherein the extraction reference determination unitdetermines the image extraction reference according to the relationshipbetween persons.
 26. The image extraction method according to claim 25,wherein the step in which the image analysis unit calculates therelationship between persons includes: a step in which an imageinformation acquisition section acquires image information regardingimages included in the first image group; a step in which an imagefeature analysis section analyzes features of the images included in thefirst image group based on the image information; and a step in which arelationship calculation section calculates a relationship betweenpersons present in the images included in the first image group based onthe image information and the image features.
 27. The image extractionmethod according to claim 26, wherein the relationship calculationsection further calculates a relationship between a person present ineach image included in the first image group and an object other thanthe person, and the extraction reference determination unit furtherdetermines the image extraction reference according to the relationshipbetween a person and an object present in each image included in thefirst image group.
 28. A non-transitory computer-readable recordingmedium on which a program causing a computer to execute each step of theimage extraction method according to claim 24 is recorded.